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Image positional relation correction apparatus, steering supporting apparatus provided with 
the tmage positional relation correction apparatus, and image positional relation correction 
method 



(57) An image positional relation apparatus is pro- 
vided, in which coordinate data of a real target photo- 
graphed by a CCD camera and coordinate data of a 
virtual target logically calculated from the predeter- 
mined position of the real target are transmitted to a 
superimpose circuit, and the real target and the virtual 
target are superimposed to be displayed on a monitor 
screen. A user operates a controller so that the real tar- 



get coincides with the virtual target. An amount of such 
operation is transmitted to the superimpose circuit as a 
correction amount, and the positional relation between 
the real target and the virtual target, i.e., the positional 
relation between the real image and the virtual image 
can be properly corrected. 
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image positional relation correction method for correcting the relative positional relation between a real image taken by 
a camera and a virtual image to be superimposed to be displayed on a monitor screen together with the real image 
comprising the steps of taking a real target by a camera, setting a virtual target theoretically derived from the coordi- 
nates of the real target on the same coordinate system as the virtual image, and correcting the relative relation between 
the real image and the virtual image based on the deviation between the real target and the virtual target. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0011] 



Fig. 1 is a block diagram illustrating an outline of an image positional relation correction apparatus in accordance 
with a first embodiment of the present invention; 

Fig. 2 is a view schematically illustrating a controller of an image positional relation correction apparatus in accord- 
ance with the first embodiment of the present invention; 

Fig. 3 is a view schematically illustrating the state in which a test chart member as real target providing means is 
mounted on the rear part of a vehicle; 

Fig. 4 is a block diagram illustrating the outline of an image positional relation correction apparatus in accordance 
with a second embodiment of the present invention; 

Fig. 5A is a side view showing the state in which an image positional relation correction apparatus in accordance 
20 with a fourth embodiment of the present invention is mounted on a vehicle; 

Fig. 5B is a plan view showing the state in which an image positional relation correction apparatus in accordance 
with the fourth embodiment of the present invention is mounted on a vehicle; 

Fig. 6A is a side view showing the state in which an image positional relation correction apparatus in accordance 
with the fourth embodiment of the present invention is mounted on a vehicle; 
25 Fig. 6B is a plan view showing the state in which an image positional relation correction apparatus in accordance 

with the fourth embodiment of the present invention is mounted on a vehicle; 

Fig. 7 is a block diagram illustrating an outline of an image positional relation correction apparatus in accordance 
with the fourth embodiment of the present invention; 

Fig. 8 is a block diagram illustrating an outline of an image positional relation correction apparatus in accordance 
30 with a fifth embodiment of the present invention; 

Fig. 9 is a block diagram illustrating an outline of an image positional relation correction apparatus in accordance 
with a sixth embodiment of the present invention; 

Fig. 1 0A illustrates a monitor screen of a conventional steering supporting apparatus in the normal state in which 
the optical axis of a lens coincides with the center of a CCD area sensor; and 
35 Fig. 1 0B illustrates a monitor screen of a conventional steering supporting apparatus in the state in which the opti- 

cal axis of the lens and the center of the CCD area sensor deviate. 

DESCRIPTION OFTHF PREFERRED EMBODIMENTS 

40 [001 2] An embodiment in which an image positional relation correction apparatus of the present invention is applied 
to the case for correcting the image positional relation between a vehicle rear image and a steering supporting guide in 
a steering supporting apparatus of a vehicle will be hereinafter described with reference to the accompanying drawings. 



First Embodiment 



[0013] Fig. 1 is a block diagram illustrating the outline of an image positional relation correction apparatus in 
accordance with a first embodiment of the present invention. A CCD camera 21 is provided with a lens 23, a CCD area 
sensor 25 and a signal processing IC 27. A signal from the signal processing IC 27 is to be inputted in a superimpose 
circuit 29. Moreover, a signal from a logic drawing circuit 31 and a correction signal from a controller 33 are also to be 
so inputted in the superimpose circuit 29. A signal from the superimpose circuit 29 is to be outputted to a monitor 35. In 
addition, as shown in Fig. 2, the controller 33 consists of a housing, and a cross button 37 capable of inputting a cor- 
rection amount in the upward, downward, leftward and rightward directions is provided on its front face. 
[0014] Operations of the image positional relation correction apparatus in accordance with this embodiment will 
now be described. First, as shown in Fig. 3, the CCD camera 21 for taking a vehicle rear image as a real image is 
mounted on the rear part of a vehicle 39. In addition, the front end of a test chart member 41 as real target providing 
means is attached to a rear bumper 39a of the vehicle 39. In this way, the test chart member 41 is disposed within an 
object area A of the CCD camera 21 . Five cross-shaped real targets 43 are drawn on the upper surface of the plate-like 
test chart member 41 . A real image including the real target 43 is taken in from the CCD area sensor 25 via the lens 
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like. In this way, even if the optical axis of the lens 23 does not coincide with the center of the CCD area sensor 25 the 
can bTm^^ Vehi ° ,e im * 9e ** * ^ ' ma9e ^ ^ su PP ortin 9 9 uide as a virtua ' ^age 



Third Embodiment 



[0022] Although the correction amount inputted in the controller 33 is transmitted to the superimpose circuit 29 in 
the first embodiment, in the third embodiment, the correction amount inputted in the controller 33 is transmitted to the 
CCD camera, for example, via a serial interface of the CCD camera, and image data correction is performed in the CCD 
camera. In such a form, as in the first embodiment, the positional relation between the vehicle rear image as a real 
image and the steering supporting guide as a virtual image can be made proper. 



Fourth embodiment 



[0023] An image positional relation correction apparatus in accordance with a fourth embodiment corrects the rel- 
ative posrtional relation between a real image and a virtual image in the case in which not only the optical axis of the 
lens does not coincide with the center of the CCD sensor area, but the CCD camera is not properly mounted on a vehi- 
cle as prescribed by the standard. 

[0024] First, the state in which the image positional relation correction apparatus in accordance with this fourth 
embod.ment is mounted on a vehicle will be described. As shown in Figs. 5A and 5B, the CCD camera 21 for taking a 
vehicle rear image is mounted on the rear part of the vehicle 39. The CCD camera 21 is provided with the lens 23 and 
the CCD area sensor 25. In addition, a monitor 65 is provided in the front of the driver's seat of the vehicle 39 
[0025] An ideal state in which the CCD camera 21 is mounted on the vehicle as prescribed by the standard and an 
optical axis 77 of the lens 23 of the CCD camera 21 coincides with the center of the CCD area sensor 25 will be 
25 described. 

[0026] For the convenience of description, a road surface coordinate system is assumed where a point on the 
ground vertically down from a rear axle center 75 of the vehicle to a road surface 76 is assumed to be an origin O a 
horizontal direction toward the rear of the vehicle 39 is assumed to be a y axis positive direction, a horizontal direction 
toward the left of the vehicle 39 is assumed to be an x axis positive direction, and a vertical direction upward of the vehi- 
30 cle 39 is assumed to be a z axis positive direction. 

[0027] The CCD camera 21 is disposed on the standard mounting position of a coordinate point PS (x y z) dis- 
played the road surface coordinate system with a standard mounting angles of a depression angle <o, a direction angle 
y and a rotation angle e. Here, the depression angle <o represents an angle downward from a y axis direction the direc- 
tion angle y represents an angle from a y axis negative direction on a surface parallel with a x-y plane, and the rotation 
angles is an angle with which the CCD camera 21 is rotated around the optical axis center 77 to be amounted 
[0028] In addition, the positional deviation amount to the x axis positive direction of the center of the CCD area sen- 
sor 25 against the optical axis 77 of the lens 23 is assumed to be ACx, and the positional deviation amount to the y axis 
positive direction is assumed to be ACy. Here, x, y, z. y, e, ACx and ACy are assumed to be standard mounting 
parameters even if they are predetermined values. 

[0029] Particularly, the optical axis 77 of the lens 23 of the CCD camera 21 and the center of the CCD area sensor 
25 are positonally adjusted in the state in which they coincide with each other. That is, both the positional deviation 
amount ACx to the x axis positive direction and the positional deviation amount Acy to the y axis positive direction of the 
CCD area sensor 25 against the optical axis 77 of the lens 23 are 0. 

^^JL ° n thS ° ther hand ' aS Sh0Wn in Rg - 6A and 6Bl if the 000 camera 21 is actual| y mounted on the vehicle 39 
the CCD camera 21 is mounted with a mounting error against the standard, and the optical axis 77 of the lens 23 of the 
CCD camera 21 does not coincide with the center of the CCD area sensor 25. 

[0031] In this case, the CCD camera 21 is disposed in the mounting position of a coordinate point PC {x+Ax y+Ay 
z+Az) represented by the road surface coordinate system with the mounting angles of the depression angle ra+Aw the 
direction angle 7+A7 and the rotation angle e+A0. 

[0032] In addition, the positional deviation amount to the x axis positive direction of the CCD area sensor 25 is ACx 
and the positional deviation amount to the y axis positive direction is ACy. 

[0033] Here, x+Ax, y+Ay, z+Az, ow-Ao* Y+Ay, 8+A8, ACx and A Cy are assumed to be actual mounting parameters 
[0034] In addition, a test chart member 81 is disposed in a predetermined position that is apart from the origin O 
on the road surface in the backward direction of the vehicle 39 by a predetermined distance. The test chart member 81 
is to be disposed in the object area A of the CCD camera 21 . In addition, a real target 83 formed of a rectangular with 
four standard points Q1 , Q2, Q3 and Q4 as vertexes is drawn on the upper surface of the test chart member 81 
[0035] Fig. 7 is a block diagram illustrating an outline of an image positional relation correction apparatus in accord- 
ance with an embodiment of the present invention. The CCD camera 21 is provided with the lens 23, the CCD area sen- 
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deviation amount ACx and ACy between the optical axis 77 and the lens 23 and the center of the CCD area sensor 25 
are assumed to be eight parameters x1 through x8 respectively, and the coordinate values of the road surface coordi- 
nate system of the standard point Q are assumed to be X Q and Y Q . 

[0045] The x, y coordinates of the image standard point P1 , the x, y coordinate of P2, x, y coordinates of P3 and 
5 the x, y coordinates of P4 are assumed to be £1 through & respectively in this order, and is represented by a general 
equation £n (provided that n is an integer number from 1 to 8). 

[0046] For example, since X Q and Y Q are given as constants by the coordinate value of the standard point Q1 the 
x coordinates of the image standard point P1 is represented as follows: 

10 £1 = F1 (xl , x2, x3, x4, x5, x6, x7, x8) 

[0047] In addition, concerning the coordinates of the image standard points P1 through P4 photographed in a real 
camera mounting state and shown on the monitor as shown in Figs. 6A and 6B, the x, y coordinates of P1 the x y coor- 
dinates of P2, the x, y coordinates of p3, and the x, y coordinates of P4 are assumed to be through £8' and is rep- 
15 resented by a general equation §n» (provided that n is an integer number from 1 to 8). 

[0048] The x coordinates and the y coordinates frl' through $8' of the image standard points P1 through P4 are 
already calculated by the controller 63. " y 

[0049] Here, if the function fn (x1 , , x8) = £n - £n' is introduced, and the values of the real mounting parameter 

being the real camera mounting state is set, fn (x1 , , x8) = 0. 

[0050] That is, if the values of the parameters x1 through x8 is calculated so that any of f 1 through f8 is 0 these 
values become the values of the real mounting parameter. 

[0051] The Newton-Raphson method is used in calculating the parameters x1 through x8. 

[0052] When the Taylor's expansion expression is introduced with respect to the function fn, approximation can be 
found as follows: 



20 



25 



30 



fn (X1 + AX1 , ... , x8 + Ax8) = fn (x1 , x8) + dfn/dxn • Axn = 0 

Therefore, 

dfn/dxn • Axn = -f n (xl x8) 

- - (£n - £n') 
= -(Fn (x1, .... x8)-£n') 
= 4n' -Fn (xl,..., x8) 

35 [0053] In addition, dfn/dxn = dFn/dxn and 

dFn/dxn • Axn = £n'-Fn(xt, .... x8) (1) 

[0054] Here, the equation (1) is represented as a matrix, the ideal standard mounting parameters as shown in Figs 
5A and 5B are substituted in xl through x8 as initial values, and dFn/dxn and Fn are calculated. Since £n' has been 
found as described above, Ax1 through Ax8 can be calculated. Then, the values of x1 through x8 are corrected by add- 
ing Ax1 through Ax8 to x1 through x8 respectively and are substituted in the equation (1) to find A xl through Ax8 By 
repeating this processing until Ax1 through Ax8 become 0 or a predetermined value in the vicinity of 0, the real mount- 
ing parameters x1 through x8 can be found. 

[0055] In this way, the real mounting parameters are calculated, and the logic drawing circuit 61 calculates new vir- 
tual target points R1 through R4 based on the real mounting parameters to display the new virtual target points R1 
through R4 on the monitor screen 65 again. In addition, since data of the virtual image, for example, display data of the 
steering supporting guide, in the same coordinate system as the virtual target 85, are prepared again, the positional 
relation between the real image and the virtual image can be properly corrected. 

[0056] With this processing, even if the optical axis 77 of the lens 23 does not coincide with the center of the CCD 
area sensor 25 and the CCD camera cannot be mounted on the vehicle properly as prescribed by the standard the 
positional relation between the real image and the virtual image can be properly corrected without physically adjusting 
the optical axis 77 of the lens 23 and without performing adjustment work for mounting the CCD camera on the vehicle 
with high accuracy as prescribed by the standard. That is, the positional relation between a vehicle rear image as the 
55 real image and a steering supporting guide as the virtual image can be made proper. 
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on the vehicle or disposed on a road surface but may be white lines drawn on the road surface. In addition, the number 
and the shape of real targets provided on the test chart member are not limrted to those shown in the above-mentioned 
embod.ments. Moreover, real target providing means is not necessarily a member separate from the vehicle like the test 
chart member, but a part of the vehicle in the object area of a camera may be assumed to be the real target 
5 [0066] In addition, although the controller 33 and the cross button 37 are used for operation in the above-mentioned 
first embodiment, and the touch panel is used in the fourth and the sixth embodiments, operating means is not limited 
to these but may be any means operable to overlap a virtual target point and an image standard point. Operation pro- 
cedures are not hmrted to the foregoing as well, but various procedures may be implemented within the scope not devi- 
ating from the purport of this invention. 
w [0067] Further, although a virtual image by logic drawing is corrected in the above-mentioned first embodiment and 
a real I . mage photographed by a CCD camera is corrected in the second embodiment, the method of correction is not 
im.ted to these but such forms may be possible as to correct the real image photographed by the CCD camera in the 
first embodiment and to correct the virtual image by logic drawing in the second embodiment 

[0068] Moreover, the above-mentioned embodiments are described concerning the case in which the embodi- 
ments are apphed to the correction of images in a steering supporting apparatus of a vehicle, an image positional rela- 
tion corrects apparatus of the present invention is not limited to this but may be applied to image correction in other 
apparatuses for superimposing to display a real image and a virtual image. 

[0069] Furthermore, if a CCD camera is mounted on a vehicle as prescribed by the standard, the Newton-Raphson 
method may be applied only to the deviation amount of the center of a CCD sensor area against the optical axis of a 
lens. On the other hand, rf the deviation amount of the center of the CCD sensor area against the optical axis of the lens 
is known or there .s no deviation amount, the Newton-Raphson method may be applied to six parameters of the posi- 
tions and the angles for mounting the CCD camera to the vehicle. 

[0070] If a deviation between a real image and a virtual image arises due to such factors as the focal length and 
skewness of a lens, and dispersion of the inclination of the light receiving surface of a CCD area sensor against the opti- 
cs cal axis of the lens, the focal length and skewness of the real lens, and the inclination of the light receiving surface of 
the CCD sensor against the optical axis of the lens may be calculated by adding the factor in parameters and applying 
the Newton-Raphson method. a 
[0071 ] In addition, although the rectangular virtual target 95 with the virtual target points R1 through R4 is used in 
the sixth embodiment, virtual targets are not limited to these but additional virtual target points R5 R6 may be used 
30 m this case, the virtual target points R1 through R4 may be shifted so that the additional virtual target points R5 R6 
overlap additional standard points Q5, Q6, ... as in the sixth embodiment, or the difference amount between the addi- 
tional v,rtual target points R5, R6, ... and the additional image standard points Q5, Q6, ... may be grasped by moving 
the R5, R6, ... directly. The amount by which the virtual target points R1 through R4 should be shifted can be calculated 
based on the grasped difference amount. In this way, since the repetition of an operator's adjustment operation, namely 
35 [shift] -> [match determination] [shift] -> [match determination], becomes unnecessary, and operation time can be 
reduced and, at the same time, correction with high accuracy becomes possible. 

[0072] Moreover, the image processing circuit used in the fifth embodiment may be added to the image positional 
relation correction apparatus of the sixth embodiment to coincide a virtual target with a real target more accurate V by 
operating the controller to the rectangular virtual target displayed as a result of the image processing 

40 [0073] Further, since the screen of the monitor displays by the unit of one dot, coordinates with the accuracy of less 
than one dot unit cannot be calculated if the virtual target has four points. However, if not only four points but also a rec- 
tangular shape is displayed as the virtual target, as the image positional relation correction apparatus of the sixth 
embodiment, the logic drawing circuit may define coordinate data of each point on the rectangular using the coordinates 
corresponding to 0.1 dot of the monitor screen. If a rectangular shape based on the coordinate data with accuracy 

45 higher than the dot of the monitor screen, the coordinates of the real target on the monitor screen can be obtained with 
higher accuracy due to the degree of linear deviation of the sides of the rectangular. 

[0074] Moreover, a virtual target may be one that not only displays a position corresponding to a real target but also 
displays a position where an optical axis of a lens of a camera reaches a road surface. 

[0075] An image positional relation apparatus is provided, in which coordinate data of a real target photographed 
so by a CCD camera and coordinate data of a virtual target logically calculated from the predetermined position of the real 
target are transmitted to a superimpose circuit, and the real target and the virtual target are superimposed to be dis- 
played on a monitor screen. A user operates a controller so that the real target coincides with the virtual target An 
amount of such operation is transmitted to the superimpose circuit as a correction amount, and the positional relation 
between the real target and the virtual target, i.e., the positional relation between the real image and the virtual imaqe 
55 can be properly corrected. ~ 
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setting a virtual target logically led from the coordinates of said real target on the same coordinate system as 
said virtual image; and 

correcting the relative positional relation between said real image and said virtual image based on the deviation 
between said real target and said virtual target 
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(54) Image positional relation correction apparatus, steering supporting apparatus provided with 
the image positional relation correction apparatus, and image positional relation correction 
method 



(57) An image positional relation apparatus is pro- 
vided, in which coordinate data of a real target photo- 
graphed by a CCD camera and coordinate data of a vir- 
tual target logically calculated from the predetermined 
position of the real target are transmitted to a superim- 
pose circuit, and the real target and the virtual target are 
superimposed to be displayed on a monitor screen. A 



user operates a controller so that the real target coin- 
cides with the virtual target. An amount of such opera- 
tion is transmitted to the superimpose circuit as a cor- 
rection amount, and the positional relation between the 
real target and the virtual target, i.e., the positional re- 
lation between the real image and the virtual image can 
be properly corrected. 
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